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Stewart-  Priven  Overview 


30+  years  software  development  Industry  experience  (each) 

•  Commercial,  Executive  Management  Focus 

•  Government,  Program  Management  &  Technical  Focus 

Managed  IBM  team  that  developed  Inspections 

Both  taught  Inspections  for  Michael  Fagan  1998  -  2005 

•  250  classes,  5,000  inspection  practitioners,  50  company  locations 


Stewart-Priven  Group  -  publications,  presentations  (www.stewart-priven.com) 

•  CrossTalk  Journal,  Jan.  2008  -  ‘How  to  Avoid  SW  Inspection  Failur e’(70  Pitfalls) 

•  CrossTalk  Journal,  Mar.  2009  -  ‘Mgt.  Insp.  Responsibility  &  Tools  for  Success’ 

•  Plenary  speakers  at  2009  Systems  &  Software  Technology  Conference 

•  Project  Mgt.  Institute/Military  Health  Systems  Oct.  2009  ‘SW  Inspection  Success’ 

•  2010  article  ‘An  Auditable  Performance  Based  SW  Acquisition  Process’ 
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Agenda 


i/ 


Government  Software  Acquisition  Problems 


4k 


A  Solution* 

Jb 


2010  article  www.stewart-priven.com 
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Errors,  Vulnerabilities,  Missed  schedules,  Reduced  content 


Focus  of  general  session  opening  at  last  year’s  SSTC  on  April  20th  2009 

Lieutenant  General  L.  William  Shelton;  U.S.  Air  Force 

-  Chief  of  Warfighting  Integration  and  Chief  Information  Officer 

-  Assistant  Vice  Chief  of  Staff  and  Director  Air  Force  Staff  Headquarters 

“  CMMI  Level  5  projects  also  experiencing  these  problems” 


Later  in  the  conference: 

Karl  Rogers  -  SSTC  host  and  Director  of  309th  Software  Maintenance  Group 

Bruce  Weimer- Army  Software  Engineering  Center,  SSTC  April  22,  2009 

•  ‘Software  Quality  Assurance,  Early  and  Continuous  throughout  the  Life  Cycle’ 

•  ‘Justifiable  evidence  and  high  confidence  that  your  system  performs  as  expected, 
when  expected,  is  safe,  and  is  secure’ 

also  addressed  these  problems 
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DoD/DHS*  SwA  Acquisition  Working  Group 


“acquisition  officials  continue  to  accept  software  riddled  with  errors  and  other 
security  vulnerabilities” 

•  The  Software  Assurance  (SwA)  Acquisition  Working  Group.  “Software  Assurance  in 
Acquisition:  Mitigating  Risk  to  the  Enterprise.”  October  22,  2008 


“Software  vulnerabilities,  malicious  code,  and  software  that  doesn’t  function 
as  promised  pose  a  substantial  risk  to  the  Nation’s  software-intensive  critical 
infrastructure  that  provide  essential  information  and  services  to  citizens” 

•  The  Software  Assurance  (SwA)  Acquisition  Working  Group.  “Software  Assurance  in 
Acquisition:  Mitigating  Risk  to  the  Enterprise.”  October  22,  2008 


DoD  -  U.S.  Department  of  Defense 

DHS  -  U.  S.  Department  of  Homeland  Security 
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Defect  Removal  Consequences 


Without  planned  early 
defect  removal  □  (typical) 
-Schedule  erodes 
-Quality  declines 
-Cost  escalates 
-Code  analyzers  not 
effective  for  Req  &  Des 


Defect  Insertion  -  Removal  -  Fix  Hours/Cost 
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With  planned  early  defect 
removal  □(e.g.,  effective 
Inspections) 

-Defect  leaks  contained 
-Quality  is  high 
-Rework  cost  minimized 
-Schedule  contained 


Defect  Insertion  -  Removal  -  Fix  Hours/Cost 
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CMM  /  CMMI  /  ISO  9x  /  etc. 


•  Predictors  of  Success 


•  Reflect  what  should  be  done  during  development, 

•  Don’t  examine  outputs  of  development  efforts 

•  Necessary,  but  not  sufficient  proof  of: 

-  What  will  be  done 

-  What  has  been  done  correctly 


Report  of  the  Defense  Science  Board  Task  Force.  “Mission  Impact  of  Foreign 
Influence  on  DoD  Software.’  Sept.  2007 

•  “Process  Assessments  by  themselves  do  not  examine  the  outputs  of  any  development 
effort  and  are  therefore  silent  with  respect  to  the  quality  attributes  of  any  particular 
product.  ” 

•  “A  positive  Process  Assessment  finding  lowers  the  risk  that  an  organization  will  produce 
a  low  quality  product  but  the  [actual]  quality  of  the  product  itself  must  be  assessed  using 
other  methods.  ” 
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SOLUTION  to  Acquiring  Software  On-Time  with  Higher  Quality 


•  Performance  Based  Software  Acquisition  -  discussed  since  1991 


•  Modified  concept  needed:  Based  upon  existing  Standards  - 

•  Concept  Overview:  IbSJ 

1 .  Candidate  suppliers  identify  specific  capabilities  during  RFP  bid  process 

•  Acquirer  (e.g.,  Govt.)  Go/No-go 

2.  Supplier  capabilities  then  verified  by  Acquirer’s  Expert  as  part  of  bid  process 

•  Acquirer  Go/No-go 

3.  Supplier  must  demonstrate  capability  to  produce  ongoing,  actionable  and 
auditable  justifiable  evidence  throughout  contract  performance 

•  Acquirer  Go/No-go  before  contract  award 

4.  Post-award  performance  monitoring,  throughout  development 


What  makes  this  concept  feasible  today? 


An  Auditable  Performance  Based  Software  Acquisition  Process 


9 


SSTC  2010 


_  Copyright©  2010  Stewart-Priven  Group,  LLC  All  Rights  Reserved 

Recently  Available  Technologies  Enabling 
Auditable  Performance  Based  Software  Acquisition 


‘IEEE  Std.  1028™-2008  for  Inspections’  (section  6) 

-  Released  August  2008 

-  Significant  upgrade  from  previous  1997  version 

-  Clarifications,  Completeness,  Inspection  Roots 


Computerized  tools  for  Inspection  Planning,  Performing,  and  Result 
Tracking  and  Measurement 

-  Topic  of  last  years  SSTC  Plenary  presentation  on  April  22nd  2009 

•  www.stewart-priven.com/publications.htm 

-  Compliant  with  ‘IEEE  1028™-2008  for  Inspections’ 

-  Provide  rigor  to  Inspection  Process  for: 

•  Correct  &  Complete  Execution 

•  Consistency  between  Inspection  teams,  organizations,  projects,  locations 

•  Repeatable  Performance 

•  Auditable  and  actionable  results,  management  reports 

-  Net  project  saving  estimate  provided  before  project  commitment 

-  ROI  and  savings  estimates  for  individual  Inspections  of  Requirements  and 
Design,  as  well  as  Code 


•  Both  technologies  target  pre-code  high  defect  insertion  points 

-  Contract,  Requirements,  Architecture,  Interfaces,  Design 
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Inspections  -  Peer  Reviews 


•  Overtime,  each  term  has  become  ambiguous 

•  Many  times  the  two  terms  are  used  interchangeably 

Stewart-Priven  believe: 

•  Inspections  are  a  rigorous  form  of  Peer  Reviews 

•  Peer  Reviews  are  not  necessarily  Inspections 

-  Peer  Reviews  may  or  may  not  be  Inspections 

•  Key  characteristics  of  effective  Inspections: 

1.  Defined  by  ‘IEEE  Std.  1 028™-2008  for  Inspections’  (section  6) 

•  Incorporate  rigorous  ‘data-based’  analysis  (initially  done  by  IBM  in  mid-70s) 

•  Limits  apply  to  material  size,  team  size,  material  rates,  Insp.  Mtg.  length 

2.  Objective  is  ‘removal’  of  major  defects 

•  not  just  finding  defects,  or  removal  of  minor  defects 

3.  Paraphrasing  by  Reader’s  role,  on  all  ‘prepared’  target  material 

4.  Real-time  team  synergism 

•  Additional  defects:  +28%  text;  +55%  code  (Michael  Fagan,  sd&m  Conference  2001) 

5.  Computerized  Inspection  tools  (for  correct,  consistent,  repeatable  execution) 

6.  Upper  management  has  implementation  responsibilities  (e.g.,  for  pitfall  avoidance) 
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‘IEEE  Std.  1028™-2008  for  Inspections’  (section  6) 


Improved  Inspection  Process  Definition  1997-2008 


25 


Should 


0  IEEE  1028™ -1997  □  IEEE  1028™-2008  □  IEEE  1028™-2008  (multipart) 


‘Shall’  (required)  ‘May’  (alternative  to  Shall)  ‘Should’  (recommended) 
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2008  Inspection  Standard  ‘Process  Actions’ 


Inspection  Objective:  Find  and  Fix  Product  Defects 

14  (pre-inspection) 


Inspection 

Process 


PLanning 
(Step  1)  ' 

Overview 
(Step  2) 

165  ‘actions’  applicable  to  Inspection  Process 

44 

> 

.  5 

_ 

PReparation 
(Step  3) 


Inspection 
Meeting 
(Step  4) 


31 


16  actions  with  multiple  Inspection  steps 


ReWork 
(Step  6) 


Follow-Up 
(Step  7) 


Verified  Product  Fixes 


8 


13  (post-inspection) 


Consistent  with  IEEE  Standard  1028™-2008  for  Inspections 
(IEEE  -  Institute  of  Electrical  and  Electronics  Engineers,  Inc.) 
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2008  Inspection  Standard  ‘Role  Actions’ 


Inspection  Objective:  Find  and  Fix  Product  Defects 

14  (pre-inspection) 


PLanning 
(Step  1)  ' 

Overview 
(Step  2) 

44 

. :  «- 

> 

5 

_ 

PReparation 
(Step  3) 

( 127)  actions 


10 


All  Inspectors 
(76) 


165  actions  applicable  to  Inspection  Process 

127  actions  applicable  to  Inspection  team 
31  actions  applicable  to  Management’s  role 
7  actions  applicable  to  Champion’s  role 


16  actions  with  multiple  Inspection  steps 


Author 


Leader 

(92)1 

Recorder 

(4) 


ReWork 
(Step  6) 


Follow-Up 
(Step  7) 


Verified  Product  Fixes 


13  (post-inspection) 


8 
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Ensuring  Supplier  Compliance  to  Inspections 


Inspection  Compliance  Matrix 


Concept: 

1.  Candidate  suppliers  identify  specific  capabilities  during  RFP  bid  process 

•  Acquirer  (e.g.,  Govt.)  Go/No-go 

2.  Supplier  capabilities  then  verified  by  Acquirer’s  Expert  as  part  of  bid  process 

•  Acquirer  Go/No-go 

3.  Supplier  must  demonstrate  capability  to  produce  ongoing,  actionable  and 
auditable  justifiable  evidence  throughout  contract  performance 

•  Acquirer  Go/No-go  before  contract  award 


4.  Post-award  performance  monitoring,  throughout  development 
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Inspection  Compliance  Matrix  -  part  1  of  4 
Parsing  the  Inspection  Standard 


line 

3/30/10 

insp 

role 

IEEE  Std.  1028™-2008  for  Inspections  (Actions) 

Action  Type 

# 

Para 

r  Action  Clarification  Text  in  brackets  1 

ShL 

May 

ShD 

1 

TOTALS  > 

165 

114 

26 

25- 

164 


165 

166 

167 

T68 

169 


170 


171 


172 


173 


174 


175 

L76 

177 

178 


6.8 


6.8 


6.8 


6.8 


6.8 


6.8 


DATA  COLLECTION 


X 


M  Inspections  shall54a  provide  data  for  the  analysis  of  the 
quality  of  the  software  product 

M  Inspections  shall54b  provide  data  for  the  effectiveness  of 

the  acquisition,  supply,  development,  operation  and  - 

maintenance  processes 

M  Inspections  shall54c  provide  data  for  the  effectiveness  and 
the  efficiency  of  the  inspection  itself 
M  data  from  the  author  and  inspectors  shal^^^NOT  be  used 
to  evaluate  the  performance  of  individuals 
L  anomalies  identified  at  an  inspection  meeting  shall56  be 
classified  in  accordance  with  6.8.1,  6.8.2,  and  6.8.3 
[anomaly  classification,  categories  and  ranking! 


4  L  Inspection  data  shall57a  contain  the  identification  of  the 
software  product 


4  D  Inspection  data  shall57b  contain  the  date  and  time  of  the 
inspection 


L  Inspection  data  shall57c  contain  the  inspection  team 


L  Inspection  data  shall57d  contain  the  preparation  and 


54a 

54b 


57a- 

57b 

57c 

57d 


inspection  times 


5  L  Inspection  data  shall57e  contain  the  volume  [size]  of  the 
materials  inspected 


L  Inspection  data  shall57f  contain  the  disposition  of  the 


57e 

57f 


_ inspected  sonware  product 

8  M  Capture  of  inspection  data  shall58  be  used  to  optimize 
_ local  guidance  for  insnections 

8  M  MapagefflEnfofinspection  data  requires  a  capability  to 
enter,  store,  access,  update,  summarize,  and  report 
classified  anomalies _ 

f  equency  and  types  of  inspection  analysis  reports,  and 
theirdisTributioiLare  left  to  local  standards  and 


58 

94 


95 


Totals 


Section  6.8  (Data  Collection)  of  Standard 


Parsing  each  shall,  may,  should 


Assigning  ID  #  to  each  shall,  may,  should 


Decomposing  multi-part  actions 


Identifying  Inspection  Role 


identifying  inspection  step  # 


Identifying  where  ‘Action’  additions  needed 
(ID#  =  9x;  e.g.,  94,  95) 
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Inspection  Compliance  Matrix  -  part  2  of  4 
Recommended  Implementation 


line 

# 

3/30/10 

insp 

step 

role 

IEEE  Std.  1028™-2008  for  Inspections  (Actions) 

Action  Type 

Action 

Change^ 

Rec. Implementation 

LJ 

n  ‘Recommended’ 

Para 

f  Action  Clarification  Text  in  brackets  1 

ShL 

May 

ShD 

Training 

Tools 

Process 

other 

Implementation 

1 

TOTALS  > 

165 

114 

26 

25 

37 

V 

82 

138 

0 

NX 

ss. 

164 

6.8  DATA  COLLECTION 

165 

6.8 

7 

M 

Inspections  shall54a  provide  data  for  the  analysis  of  the 
quality  of  the  software  product 

54a 

54b 

54c 

55 

56 

57a 

57b 

57c 

57d 

57e 

57f 

58 

94 

X 

X 

X 

V  X 

166 

6.8 

7 

M 

Inspections  shall54b  provide  data  for  the  effectiveness  of 
the  acquisition,  supply,  development,  operation  and 
maintenance  processes 

X 

x\ 

167 

6.8 

5 

M 

Inspections  shall54c  provide  data  for  the  effectiveness  and 
the  efficiency  of  the  inspection  itself 

X 

X 

168 

6.8 

8 

M 

data  from  the  author  and  inspectors  shall55  NOT  be  used 
to  evaluate  the  performance  of  individuals 

X 

X 

169 

6.8 

4 

L 

anomalies  identified  at  an  inspection  meeting  shall56  be 
classified  in  accordance  with  6.8.1,  6.8.2,  and  6.8.3 

Tan omaly  classification,  categories  and  ranking] 

•  Enhancements 

•  Most  are  text 
clarifications 

170 

6.8 

4 

L 

Inspection  data  shall57a  contain  the  identification  of  the 
software  product 

X 

X 

X 

X 

X 

X 

X 

X 

171 

6.8 

4 

D 

Inspection  data  shall57b  contain  the  date  and  time  of  the 
inspection 

X 

X 

172 

6.8 

4 

L 

Inspection  data  shall57c  contain  the  inspection  team 

X 

X 

173 

6.8 

4 

L 

Inspection  data  shall57d  contain  the  preparation  and 
inspection  times 

X 

X 

174 

6.8 

5 

L 

Inspection  data  shall57e  contain  the  volume  [size]  of  the 
materials  inspected 

X 

X 

175 

6.8 

5 

L 

Inspection  data  shall57f  contain  the  disposition  of  the 
inspected  software  product 

X 

X 

176 

6.8 

8 

M 

Capture  of  inspection  data  shall58  be  used  to  optimize 
local  guidance  for  inspections. 

X 

X 

177 

6.8 

8 

M 

Management  of  inspection  data  requires  a  capability  to 
enter,  store,  access,  update,  summarize,  and  report 
classified  anomalies 

add  a  shall 

X 

X 

178 

6.8 

8 

M 

Frequency  and  types  of  inspection  analysis  reports,  and 
their  distribution,  are  left  to  local  standards  and 

95 

add  a  should 
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Inspection  Compliance  Matrix  -  part  3  of  4 
Supplier  provided  Implementation 


1028-2008 


Supplier  Map 


Insp.  Expert 


A-Author  C-Champion  D-RecorDer  I-Inspectors  L-Leader  M-Management  R-Reader(paraphraser)  0-pre-inspect  8-post-inspect 


line 

3/30/10 

insp 

role 

IEEE  Std.  1 028™-2008  for  Inspections  (Actions) 

Action  Type 

Action 

Rec. Implementation 

Supplier  Implementation 

# 

Para 

r  Action  Clarification  Text  in  brackets  1 

ShL 

May 

ShD 

Change 

Training 

Tools 

Process 

other 

Training 

Tools 

Process 

other 

none 

1 

TOTALS  > 

165 

114 

26 

25 

37 

139 

82 

138 

0 

0 

0 

0 

0 

0 

164 

6.8 

DATA  COLLECTION 

165 

6.8 

7 

M 

Inspections  shall54a  provide  data  for  the  analysis  of  the 
quality  of  the  software  product 

54a 

X 

X 

X 

166 

6.8 

7 

M 

Inspections  shall54b  provide  data  for  the  effectiveness  of 
the  acquisition,  supply,  development,  operation  and 
maintenance  processes 

54b 

X 

X 

X 

167 

6.8 

5 

M 

Inspections  shall54c  provide  data  for  the  effectiveness  and 
the  efficiency  of  the  inspection  itself 

54c 

X 

X 

X 

168 

6.8 

8 

M 

data  from  the  author  and  inspectors  shall55  NOT  be  used 

55 

to  evaluate  the  performance  of  individuals 

169 

6.8 

4 

L 

anomalies  identified  at  an  inspection  meeting  shall56  be 
classified  in  accordance  with  6.8.1,  6.8.2,  and  6.8.3 
f anomaly  classification,  categories  and  rankingl 

56 

170 

6.8 

4 

L 

Inspection  data  shall57a  contain  the  identification  of  the 

57a 

software  product 

171 

6.8 

4 

D 

Inspection  data  shall57b  contain  the  date  and  time  of  the 

57b 

inspection 

172 

6.8 

4 

L 

Inspection  data  shall57c  contain  the  inspection  team 

57c 

X 

X 

X 

173 

6.8 

4 

L 

Inspection  data  shall57d  contain  the  preparation  and 
inspection  times 

57d 

X 

X 

X 

174 

6.8 

5 

L 

Inspection  data  shall57e  contain  the  volume  [size]  of  the 

57e 

materials  inspected 

175 

6.8 

5 

L 

Inspection  data  shall57f  contain  the  disposition  of  the 

57f 

inspected  software  product 

176 

6.8 

8 

M 

Capture  of  inspection  data  shall58  be  used  to  optimize 

58 

local  guidance  for  inspections. 

177 

6.8 

8 

M 

Management  of  inspection  data  requires  a  capability  to 
enter,  store,  access,  update,  summarize,  and  report 
classified  anomalies 

94 

add  a  shall 

X 

178 

6.8 

8 

M 

Frequency  and  types  of  inspection  analysis  reports,  and 

95 

add  a  should 

their  distribution,  are  left  to  local  standards  and 

Legend: 

•  Standard 

•  Supplier 

•  Expert 

•  Roles 


Supplier  provided 
Implementation 
capability 


5th  column  added 

Mr 
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Inspection  Compliance  Matrix  -  part  4  of  4 
Action  Cross-Reference 


Action  Cross 
Reference 


|  1028-2008  |  ^  ^  'insp.  Expert^H 


A-Author  C-Champion  D-RecorDer  I-Inspectors  L-Leader  M-Management  R-Reader(paraphraser)  0-pre-inspect  8-post-iiispect 


line 

# 

3/30/10 

insp 

step 

role 

IEEE  Std.  1G28™-2008  for  Inspections  (Actions) 

Action  1 

fype 

Action 

Change 

Rec.  Implementation 

Sup 

plier  Implementation 

Action  X-REF 

and  Notes 

08=65/23/22,  97=46/15/28 

Para 

f  Action  Clarification  Text  in  brackets  1 

ShL 

May 

ShD 

Training 

Tods 

Process 

other 

Training 

Tods 

Process 

other 

none 

1 

TOTALS  > 

165 

114 

26 

25 

37 

139 

82 

138 

0 

0 

0 

0 

0 

0 

164 

6.8  DATA  COLLECTION 

165 

6.8 

7 

M 

Inspections  shall54a  provide  data  lor  the  analysis  of  the 
quality  of  the  software  product 

54a 

54b 

54c 

55 

56 

57a 

57b 

57c 

57d 

57e 

57f 

58 

94 

X 

X 

X 

X 

X 

ref  Mandatory  2 

166 

6.8 

7 

M 

Inspections  shall54b  provide  data  for  the  effectiveness  of 
the  acquisition,  supply,  development,  operation  and 
maintenance  processes 

X 

X 

ref  Mandatory  2 

167 

6.8 

5 

M 

Inspections  shall54c  provide  data  for  the  effectiveness  and 
the  effidaicy  of  the  inspection  itself 

X 

X 

ref  May  1 

168 

6.8 

8 

M 

data  from  the  author  and  inspectors  shall55  NOT  be  used 
to  evaluate  the  perfonnance  of  individuals 

X 

X 

169 

6.8 

4 

L 

anomalies  identified  at  an  inspection  meeting  shall56  be 
classified  in  accordance  with  6.8. 1, 6.8.2,  and  6.8.3 
[anomaly  classification,  categories  andrankingl 

170 

6.8 

4 

L 

Inspection  data  shall57a  contain  the  identification  of  the 
software  product 

X 

X 

X 

X 

X 

X 

X 

X 

ref  shall  53d 

171 

6.8 

4 

D 

Inspection  data  shall57b  contain  the  date  and  time  of  the 
inspection 

X 

X 

172 

6.8 

4 

L 

Inspection  data  shall57c  contain  the  inspection  team 

X 

X 

ref  shall  53b 

173 

6.8 

4 

L 

Inspection  data  shall57d  contain  the  preparation  and 
inspection  times 

X 

X 

ref  shall  53c 

174 

6.8 

5 

L 

Inspection  data  shall57e  contain  the  volume  [size]  of  the 
materials  inspected 

X 

X 

ref  shall  53e 

175 

6.8 

5 

L 

Inspection  data  shall57f  contain  the  disposition  of  the 
inspected  software  product 

X 

X 

ref  shall  53i 

176 

6.8 

8 

M 

Capture  of  inspection  data  shall58  be  used  to  optimize 
local  guidance  for  inspections. 

X 

X 

177 

6.8 

8 

M 

Management  of  inspection  data  requires  a  capability  to 
enter,  store,  access,  update,  summarize,  and  report 
classified  anomalies 

add  a  shall 

X 

X 

178 

6.8 

8 

M 

Frequency  and  types  of  inspection  analysis  repots,  and 
their  distribution,  are  left  to  local  standards  and 

95 

add  a  should 
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3-Stage  /  8-Step  Auditable  Performance  Based  SW  Acquisition  Process 


Initial  Capability  Assessment 
(Stage  1 ) 


1 


Process  Assessment 
(Stage  2) 


Require  IEEE  Std.  1028™- 
2008  (sec. 6)  Inspection 
compliance  during 
Acquisition  proposal  bid 
response 


Provide  Inspection 
Compliance  Matrix  to 
supplier  bidders 

i  Perform  gap  analysis  and 
|  map  project’s  Inspection  (or 
I  Peer-Review)  capabilities  to 
l  Cgm  pi  ia  nce_Ma_tri_x_ 

Evaluate  mapping  and 
Recommend  Go/No-Go 


Evaluation  of  Supplier 
Inspection  Process  for: 

Educated  Management 
•Ensure  all  inspection 

pitfalls1  mitigated 

r - i 

|  Trained  Inspectors  ! 
l _ i 


Computerized 
Inspection  Tools  2 


n 

I 

I 

I 


I 
I 

L  _ . 

Go/No-Go 
Recommendation 


^^Go  -  process  verified 


Og°  -  capabilities  mapped 


8 


Execution  Assessment 
(Stage  3) 


IEEE  Std.  1028™-2008 
Compliant  Inspection  process 

executk>_n_ 

Auditable  &  Actionable 
performance-based  Results 
captured  by  Inspection-Tool 
reports 


O  Go  -  execution  confirmed 

contract  awarded  -  performance 


Monitor  Inspection  tool  reports 
for  process  conformance  and 
action  completion  throughout 
Development 

Provide  periodic  assessment 
recommendations  to  Acquirer 


Disciplined  Development  Process  (Inspection  Std.) 

Legend: 

Acquirer 

|  Supplier  | 

Acquirer’s  3rd  party  expert 

Meaningful  Metrics 

(Inspection  Tools) 


1  Stewart,  Roger  &  Priven,  Lew.  “How  to  Avoid  Software  Inspection  Failure  and  Achieve  Ongoing  Benefits.”  CROSSTALK  Magazine  Jan.  2008 


2  Stewart,  Roger  &  Priven,  Lew.  “Management’s  Inspection  Responsibilities  and  Tools  for  Success.”  CROSSTALK  Magazine  Mar/Apr.  2009 
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Capability  Mapped  -  Process  Verified  -  Execution  Confirmed 


Acquirer’s  checklist  (pre-contract  award) 


line 

# 

3/30/10 

insp 

step 

role 

IEEE  Std.  1 028™-2008  for  Inspections  (Actions) 

Action  Type 

Action 

Change 

Rec.lmplementation  ] 

Supplier  Implementation  i 

1 

M 

a 

P 

2 

V 

e 

r 

3 

C 

0 

n 

Para 

[  Action  Clarification  Text  in  brackets  1 

ShL 

May 

ShD 

Training 

Tools 

Process 

other  1 

Training 

Tools 

Process 

other 

nonet 

1 

TOTALS  > 

165 

114 

26 

25 

37 

139 

82 

138 

0  t 

0 

0 

0 

0 

0  [ 

164 

6.8  DATA  COLLECTION 

- r 

165 

6.8 

7 

M 

Inspections  shall54a  provide  data  for  the  analysis  of  the 
quality  of  the  software  product 

54a 

54b 

54c 

55 

X 

X 

X 

X 

X 

1 

fi 

166 

6.8 

7 

M 

Inspections  shall54b  provide  data  for  the  effectiveness  of 
the  acquisition,  supply,  development,  operation  and 
maintenance  processes 

X 

X 

1 

1 

167 

6.8 

5 

M 

Inspections  shall54c  provide  data  for  the  effectiveness  and 
the  efficiency  of  the  inspection  itself 

X 

X 

i 

i 

168 

6.8 

8 

M 

data  from  the  author  and  inspectors  shall55  NOT  be  used 
to  evaluate  the  performance  of  individuals 

X 

X 

1 

169 

6.8 

4 

L 

anomalies  identified  at  an  inspection  meeting  shall56  be 
classified  in  accordance  with  6.8.1,  6.8.2,  and  6.8.3 

I  anomaly  classification,  categories  and  ranking! 

56 

57a 

57b 

57c 

57d 

57e 

57f 

58 

94 

lr 

1 

i 

170 

6.8 

4 

L 

Inspection  data  shall57a  contain  the  identification  of  the 
software  product 

X 

X 

X 

X 

X 

X 

X 

X 

t 

1 

171 

6.8 

4 

D 

Inspection  data  shall57b  contain  the  date  and  time  of  the 
inspection 

X 

X 

1 

I 

172 

6.8 

4 

L 

Inspection  data  shall57c  contain  the  inspection  team 

X 

X 

1 

173 

6.8 

4 

L 

Inspection  data  shall57d  contain  the  preparation  and 
inspection  times 

X 

X 

1 

1 

174 

6.8 

5 

L 

Inspection  data  shall57e  contain  the  volume  [size]  of  the 
materials  inspected 

X 

X 

1 

1 

175 

6.8 

5 

L 

Inspection  data  shall57f  contain  the  disposition  of  the 
inspected  software  product 

X 

X 

1 

1 

176 

6.8 

8 

M 

Capture  of  inspection  data  shall58  be  used  to  optimize 
local  guidance  for  inspections. 

X 

X 

1 

177 

6.8 

8 

M 

Management  of  inspection  data  requires  a  capability  to 
enter,  store,  access,  update,  summarize,  and  report 
classified  anomalies 

add  a  shall 

X 

X 

1 

1 

1 

178 

6.8 

8 

M 

Frequency  and  types  of  inspection  analysis  reports,  and 
their  distribution,  are  left  to  local  standards  and 

95 

add  a  should 

1 
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Computerized  Inspection  Tools 


Correct  &  Complete  Inspection  Execution 


30 


Repeatable  Results  for  Labor  Savings  &  High  Quality  Products  >  (Q 


Consistency  across  Inspection  Teams,  Groups  &  Locations 


J 


Measurement  and  Comparison  of  actual  defect  removal  by  Inspection 
and  Testing  vs.  Quality  Plan  objectives 

Facilitates  Management  Buy-in 

-  Inspection  Tools  for  Project  Planning  and  Savings  Estimation 

•  Pre-Commitment 

•  Support  ‘What-lf  Project  scenarios 
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Portability  of  8-Step  Auditable  Acquisition  Process 


Could  be  applied  to  other  Standards  or  Process 

•  Standard/Process  Expert 

•  Compliance  Matrix  Development 
Matrix  Compliance  provides; 

•  Supplier  Execution  Rigor 

•  Auditable  Performance  Based  Results  from  Supplier 

•  e.g.,  tool  generated 


Inspections  can  be  used  to  examine  other  Standards  and  Processes 
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Stewart-Priven  Group  -  Contact  Information 


Address 

Stewart-Priven  Group,  LLC 
7962  Old  Georgetown  Road,  Suite  B 
Bethesda  MD  20814  USA 

Phone 

865-458-6685 


Fax 


865-458-9139 


Email 

info@stewart-priven.com 

Web  Site 

www.stewart-priven.com 


STEWART-PRIVEN  GROUP 

Software  Inspection 
‘Methodology’  Experts 


-voiding  Inspection  Pitfa//^ 


with 

SpectorTool 
Suite  ™ 


PlanSpector  ™ 
Inspector  ™ 
TrackSpector  ™ 


Development  Infrastructure 


1.1  Net  Savings 
Estimate  from 
Inspections 

1.2  Development 
Infrastructure 
Assessment 


1 


1.3.  Tuning 

Recommendation; 
&  any  Prerequisite 
Infrastructure 
Implementation 


Assessment  Methodology 


Stewart-Priven 

Methodology 


Inspection  Tools  & 
Process  Training 


2.  Management 
Instruction 

3.  Practitioner 
Training 


Implementation 


~L 


4.  Performance 
Consulting 
and  Coaching 


www.stewart-priven.com 


VI  .5 
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What  is  the  Industry  View  of  Inspections 


The  data  in  support  of  the  quality,  cost  and  schedule  impact  of  inspections  is  overwhelming. 
They  are  an  indispensable  part  of  engineering  high-quality  software.’  Steve  McConnell  -  “IEEE 
Software  Jan/Feb  2000,  Best  Influences  on  Software  Engineering  over  past  50  years” 

‘Inspections  are  surely  a  key  topic,  and  with  the  right  instrumentation  and  training  they  are  one  of  the 
most  powerful  techniques  for  defect  detection.  They  are  both  effective  and  efficient,  especially  for 
up-front  activities.  In  addition  to  large-scale  applications,  we  are  applying  them  to  smaller 
applications  and  incremental  development.’  Chris  Ebert  -  “IEEE  Software  Jan/Feb  2000,  Best 
Influences  on  Software  Engineering  overpast  50  years” 

‘Inspection  repeatedly  has  been  demonstrated  to  yield  up  to  a  10  to  1  return  on  investment.  .  . 
.depressingly  few  practitioners  know  about  the  30  year  old  technique  of  software  inspection.  Even 
fewer  routinely  perform  effective  inspections  or  other  types  of  peer  reviews.’  “Karl  Wiegers  -  “The 
More  Things  Change,  Better  Software,  Oct.  2006” 

The  software  community  has  used  Inspections  for  almost  twenty  eight  years.  During  this  timeframe 
Inspections  have  consistently  added  value  for  many  software  organizations.  Yet  for  others, 
Inspections  never  succeeded  as  well  as  expected,  primarily  because  these  organizations  did  not 
learn  how  to  make  Inspection  both  effective  and  low  cost.’  Ron  Radice  -  “High  Quality  Low  Cost 
Software  Inspections,  2002  Paradoxicon  Publishing” 

'Formal  inspections  can  raise  the  [defect]  removal  efficiency  to  over  95%.  But  part  of  the 
problem  here  is  that  not  a  lot  of  companies  know  how  to  use  these  things.'  Capers  Jones,  Chief 
Scientist,  SPR  -  "Computer  Aid  Inc.  July  2005" 

‘I  continue  to  be  amazed  at  the  number  of  software  development  organizations  that  do  not  use  this 
powerful  method  [inspections]  to  improve  quality  and  productivity.’  Ed  Weller  -  “Jan.  2002, 

Calculating  the  Economics  of  Inspections” 
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About  Stewart-Priven 


Roger  Stewart  is  co-founder  and  Managing  Director 
of  the  Stewart-Priven  Group.  He  is  an  experienced 
Lead  Systems  Engineer  and  Program  Manager  in 
both  government  and  commercial  system 
development  -  including  Systems  Engineering, 
Software  Development,  System  Integration,  System 
Testing,  and  Process  Improvement. 

Previously,  Stewart  taught  the  Fagan  Defect-Free 
Process  for  Michael  Fagan  Associates  (8  years) 
after  spending  31  years  with  IBM’s  Federal  Systems 
Division,  (now  part  of  Lockheed-Martin)  managing 
and  developing  systems  for  the  FAA  Air  Traffic 
Control,  Air  Force  Satellite  Command  &  Control, 
NASA  On-Board  Space  Shuttle,  NAVY  Light 
Airborne  Multi-Purpose  System  (LAMPS 
Helicopter);  and  in  Commercial  Banking, 
Telecommunication  and  Networking  systems. 

Roger  has  a  BS  in  Mathematics  from  Cortland 
University. 


Lew  Priven  is  co-founder  and  Managing  Director  of 
the  Stewart-Priven  Group.  He  is  an  experienced 
executive  with  management  and  technical 
background  in  system  and  software  development, 
software  quality  training,  management  development 
training  and  human  resource  management. 

Previously,  Priven  managed  the  IBM  team  that 
developed  the  inspection  process,  taught  the  Fagan 
Defect-Free  Process  for  Michael  Fagan  Associates 
(8  years),  and  was  Vice-President  of  Engineering  & 
Application  Development  at  General  Electric 
Information  Services,  Vice  President  of  Application 
Development  for  IBM’s  Application  Systems 
Division,  Director  of  Operations  &  Development  for 
the  IBM  Information  Network,  Vice  President  of 
Information  Technology  &  Human  Resources  for 
Satellite  Business  Systems. 

Lew  has  a  BS  in  Electrical  Engineering  from  Tufts 
University  and  an  MS  in  Management  from 
Rensselaer  Polytechnic  Institute. 
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Acronyms 

#  -  number 

Para  -  paragraph 

APV  -  approval 

Rec  -  Recommended 

CMM  -  Capability  Maturity  Model 

Req  -  Requirements 

CMMI  -  Capability  Maturity  Model  Integration 

RFP  -  Request  for  Proposal 

Con  -  confirmed 

ROI  -  Return  on  Investment 

Des  -  Design 

ShD  -  should 

DHS  -  Department  Homeland  Security 

ShL  -  shall 

DoD  -  Department  of  Defense 

SSTC  -  Systems  &  Software  Technology 

e.g.  -  for  example 

Conference 

Govt.  -  Government 

Std.  -  Standard 

IBM  -  International  Business  Machines 

SW  -  Software 

IEEE  -  Institute  of  Electrical  &  Electronic 

SwA  -  Software  Assurance 

Engineers,  Inc. 

TRW  -  defense  contractor  acquired  by 

Insp.  -  Inspection 

Northrop  Grumman  in  2002 

ISO  -  International  Organization  for  Standardization 

ut  -  unit  test 

Mgt  -  Management 

Ver  -  Verified 

Mtg  -  Meeting 

vs.  -  versus 

X-Ref  -  Cross  Reference 
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